April 13, 2002 


TO: Jim Wilson 

FROM: Don Barnes 

subject: EPA’s Science Advisory Board and peer review 

Thanks for the opportunity to comment on SAB vis a vis the upcoming SRA 
conference on Peer Review. There is nomenclature confusion within the Agency, so it is 
not surprising that there is confusion outside the Agency, as well. 

Peer Review and Peer Involvement 

As described in the Agency's Peer Review Handbook , the Agency makes a distinction 
between Peer Review and Peer Involvement. 

Peer Review is the outside, independent, critical review with which we are all 
familiar. It usually occurs at the end of the process; i.e., after the Agency has examined a 
scientific issue and has developed a position upon which it will construct its policy 
position. For example, for the NAAQS, the Agency generates a massive Criteria 
Document and brings it to the SAB (CASAC, specifically) for critical peer review; i.e., 
"Did we find all of the science and have we assessed it appropriately?" 

Once the SAB feels that the Criteria Document is in good shape, the Agency goes off 
and develops a Staff Paper that describes how the Agency will take the science in the 
Document and use it to develop a policy position; i.e , "And the permitted level of ozone 
will be..." The Staff Paper is brought to the SAB who are asked "Is the Agency’s 
position (which is constructed on the scientific foundation that you approved) consistent 
with the science? That is, has the science been used in an arguably appropriate manner?" 
The SAB is not being asked to agree with - or to pick — a particular regulatory level. 
Rather, they are being asked whether or not the Agency's position is credible, In light of 
the science. 

Note that the review of the Staff Paper is not "peer review" in the classic sense. 
Rather, it is more of a "laugh test" applied to the Agency’s use of the science. This 
statement is not meant to demean the Staff Paper peer review; rather, just as it is an 
important peer review function to be able to attest to the validity of the science going into 
a decision, it is equally important for peers to agree that the science was not misused or 

abused. 

Peer Involvement is used to refer to a range of activities that occur early in the 
development of the Agency's science and the construction of the policy position that is 
built upon it. Peer involvement can include such things as 

a) The Agency's giving talks about the evolving science positions at professional 
scientific meetings and engaging in subsequent discussion. 
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b) The Agency's conducting a public workshop, soliciting input from the outside 
scientific community on the technical issues associated with the problem at hand; 
e.g., the 1993 SRA workshop in San Diego on aspects of the cancer guidelines. 

c) The Agency's commissioning papers on specific aspects an issue; e.g., the writing 
of chapters for the Dioxin Reassessment. 

In going from examples a-c, the involvement from the non-EPA peer becomes more 
substantial and, hence, that particular peer becomes less appropriate to be a part of the 
independent peer review that assesses the Agency's final document. 

The Range of SAP Activities 

The SAB performs both Peer Review and Peer Involvement functions. 

a. SAB Peer Review 

1. SAB Reviews 

The meat-and-potatoes of the SAB's meal is classic peer review of near-final Agency 
scientific (not policy) documents. There is always a Charge; i.e., a specific set of 
questions that the Agency wants the SAB to answer during the course of its review. 

These questions guide, but do not constrain, the SAB in its deliberations and in its 
reports. The SAB deliberates in public and eventually issues its view in a formal written 
report that goes to the Administrator. 

2. SAB Advisories 

An SAB Advisory is very similar to an SAB review, but it is applied to an Agency 
product which that is still evolving within EPA. The subjects of Advisories are usually 
topics that will be years in the making; e.g., the National Air Toxics Assessment (NATA) 
program that was the subject of an SAB Advisory last year. Following a two-day public 
meeting, a written report was prepared and submitted to the Agency. The Agency, in 
turn, takes the Advisory under ~ yes, advisement:) - and eventually come forward with a 
final document that the SAB will again review in a similar formal process. In order to 
insure that that "second-SAB-bite-at-the-appIe" is independent and not totally sold out 
to the suggestions in the Advisory, the SAB endeavors to insure that there is a significant 
number of participants on the SAB final review panel who are different from those who 
served on the panel that wrote the Advisory. This process of bringing new faces on 
board for the final review is a point of some contention with the Agency, since the 
Agency would naturally prefer not to nw the risk of having some "newbees" come in and 
give them totally different advice. In practice, this fear is rarely manifest. 

SAB Peer Involvement 

1. SAB Commentaries 

An SAB Commentary is a written document that is essentially "unsolicited advice” on 
a topic that the Board feels should be brought to the Agency’s attention. Some people in 
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the Agency view Commentaries as "the SAB's going off the ranch again"! Others value 
the fresh perspective on an issue that may have escaped the Agency's attention. A classic 
example of an SAB Commentary is one written in 1989 about the need for the Agency - 
and the other government agencies — to get its act together in regard to computer models 
(e.g., fate-and- transport, risk assessment extrapolation, PBPK, etc,) In the Board's mind, 
there was a proliferation of models with little attention being paid to a wide range of 
issues including verification, validation, appropriate use of models, choosing among 
competing models, characterizing uncertainties in the results of models, dealing with 
proprietary models, articulating limitations in the models to the unwary, etc. When the 
Agency's efforts to address these problems faltered in the mid-1990s, the SAB issued 
another Commentary on Models, reiterating and expanding on the concerns. In response, 
the Agency conducted a productive workshop ("Models 2000") that moved the Agency 
further in the right direction. 

SAB Commentaries are less effective when there is no "customer/champion" for them 
in the Agency. The Agency already has too many things to do now and cannot follow up 
on every good idea that some of the best scientific minds in the country can dream up, so 
the Board needs to be careful to align with its customers — even if it takes some nudging 
to get them to see the right line.:) 

2. SAB Workshop reports 

In recent years, the SAB has begun sponsoring (or co-sponsoring) public workshop 
designed to spotlight certain issues. For example, the SAB co-sponsored a workshop last 
year that examined how the community around Tampa Bay made policy decisions that 
impacted the ecology of the Bay. The goal was to explore different perspectives — 
particularly those of the non-economic social sciences; e g., social anthropology — on the 
issue. The result of the public meeting was simply a factual report of events at the 
workshop, but it did not reach conclusions on the issues nor does it represent a formal 
SAB position. 

3. SAB Consultations 

The SAB has been aggressively seeking to become involved in the development of 
Agency positions very early in the process. The SAB Consultation is a format developed 
to achieve this end. The Agency approaches the SAB for a Consultation before the 
Agency’s a pproach to the topic — let alone a draft position — has been determined. The 
Consultation itself is a public meeting in which the Agency describes its problem and the 
options that it is considering as it moves forward. The SAB panel members, as 
individuals, then weigh in with their ideas, opinions, perspectives, etc. There is no 
attempt or intention to reach a consensus of the group. Rather than seeking a common 
position about what the Board things, the Consultation' is designed to leaven the Agency's 
thinking with a range of fresh — competing or even incompatible — ideas from smart 
individuals on the panel. Some of the best Consultations are those in which the members 
of the panel clearly disagree and forcefully argue their respective positions. The Agency, 
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listening in the room along with the public, is the beneficiary of the debate. There is no 
formal report resulting from a Consultation. Hence, the SAB does not "speak”, per se. 
The Board simply writes a generic "Notification of an SAB Consultation" letter to the 
Administrator stating that the Board met with the Agency and shared views on the issue. 
To the extent that those views become a matter of record, they are captured by Agency 
personnel and the public in the room, and they are reflected in the minutes of the meeting. 
Those views, however, are pointedly not recorded in the letter to the Administrator, test 
someone interpret a comment in the letter as being an SAB position. 

Some confusion arises in the case of Consultations because the same term is used in 
other contexts with a somewhat different meaning. For example. Congress, in the Safe 
Drinking Water Act, directs the Agency to "consult" with the SAB at some point during 
the development of its position. Since the SAB, per se, does not speak in an SAB 
Consultation, the intent of Congress appears to be that the Board conduct a Review or an 
Advisory in such circumstances, 

Conclusion 

The SAB, as a group of 100 members and 300+ consultants — none of whom is a 
Federal employee — plays a vital role in assisting the Agency to develop positions {e.g., 
regulations, research plans, etc.) that are built on a foundation of sound science. To carry 
out its broad mandate, the SAB has evolved a series of activities that include both peer 
review and peer involvement. In doing so, the Board has increased the credibility of 
Agency actions, helped to distinguish risk assessment (science) issues from risk 
management (policy) issues, and made the decision-making process more open to public 

scrutiny. 


4 


PM3001241829 


Source: https://www.industrydocuments.ucsf.edu/docs/lndk0001 




